Method and device for processing product manufacturing messages, electronic device, and computer-readable storage medium

ABSTRACT

A method and a device for processing product manufacturing messages, and an electronic device are disclosed. The method for processing product manufacturing messages includes: monitoring a plurality of product manufacturing messages; establishing a product defect analysis task queue based on the plurality of product manufacturing messages; distributing product defect analysis tasks to product manufacturing assisting devices based on the product defect analysis task queue, wherein the product defect analysis tasks include a task of identifying product defect content based on a defect identification model; wherein the product defect content includes any one or more of: product defect type, product defect location, and product defect size.

TECHNICAL FIELD

The embodiments of the present disclosure relate to a method and adevice for processing product manufacturing messages, an electronicdevice, and computer-readable storage medium. The present disclosurealso relates to artificial intelligence field and big data field,specifically to a system and a method for assisting productmanufacturing, and computer-readable storage medium.

BACKGROUND

In the product manufacturing procedure, for example, in themanufacturing procedure of semiconductor products, due to the problemsin devices, parameters, operations, environments, and the like, theproduced products may not meet the process requirements or even lead todefects. Therefore, it is necessary to immediately calculate andidentify defect types, defect sizes, defect positions, and otherinformation of defective products that do not meet the requirementsafter each process, and make timely correction and improvement to avoidthe continuing occurrence of defects. Currently, traditional methods fordefect identification mainly rely on manual detection, which requiresprofessional training for inspectors, especially in the case of multipleproduct models and complex problems. For example, semiconductor productshave various types of defects, which may include particle, remain, weakline, hole, splash, electrostatic breakdown, wrinkle, film color,bubble, and the like. It requires a long and dedicated time andattention from inspectors to find defects and make relevant judgments.In summary, the traditional methods for defect identification haveproblems of low efficiency and low accuracy.

During intelligent product manufacturing, a large number of productmanufacturing messages are generated. These product manufacturingmessages can be used to indicate the manufacturing process of a product,or to indicate a possible defect of the product in the manufacturingprocedure. For example, in the manufacturing procedure of semiconductorproducts, due to the problems in devices, parameters, operations,environments, and the like, the produced products may not meet theprocess requirements or even lead to defects. Therefore, it is necessaryto immediately calculate and identify defect types, defect sizes, defectpositions, and other information of defective products that do not meetthe requirements after each process, and make timely correction andimprovement.

Currently, the processing of product manufacturing messages, especiallythose about product defects, still suffers from low processingefficiency. Also, the current processing of product manufacturingmessages is still not well coordinated with the product manufacturingprocedure, thereby causing inconvenience to product manufacturing.

SUMMARY

A method for processing product manufacturing messages is providedaccording to at least one embodiment of the present disclosure. Themethod for processing product manufacturing messages comprises:monitoring a plurality of product manufacturing messages; establishing aproduct defect analysis task queue based on the plurality of productmanufacturing messages; distributing product defect analysis tasks toproduct manufacturing assisting devices based on the product defectanalysis task queue, wherein the product defect analysis tasks comprisesa task of identifying product defect content based on a defectidentification model; wherein the product defect content includes anyone or more of product defect type, product defect location, and productdefect size.

An electronic device is provided according to at least one embodiment ofthe present disclosure. The electronic device comprises a processor; anda memory storing computer instructions that, when executed by theprocessor, implement the method above.

A computer-readable storage medium with instructions stored thereon isprovided according to at least one embodiment of the present disclosure.When the instructions are executed by a processor, the method above isimplemented.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical scheme of at least oneexample embodiment of the present disclosure, the following is a briefdescription of the drawings required to be used in the description ofthe example embodiment. The drawings in the following description aremerely exemplary embodiments of the present disclosure.

FIG. 1 is a schematic diagram illustrating an example scenario forprocessing product manufacturing messages.

FIG. 2 is a flowchart illustrating a method for processing productmanufacturing messages according to at least one embodiment of thepresent disclosure.

FIG. 3 is another flowchart illustrating the method for processingproduct manufacturing messages according to at least one embodiment ofthe present disclosure.

FIG. 4 is a schematic diagram illustrating the method for processingproduct manufacturing messages according to at least one embodiment ofthe present disclosure.

FIG. 5 is another flowchart illustrating the method for processingproduct manufacturing messages according to at least one embodiment ofthe present disclosure.

FIG. 6 is another schematic diagram illustrating the method forprocessing product manufacturing messages according to at least oneembodiment of the present disclosure.

FIG. 7 is another schematic diagram illustrating the method forprocessing product manufacturing messages according to at least oneembodiment of the present disclosure.

FIG. 8 is a schematic diagram illustrating a device for processingproduct manufacturing messages according to at least one embodiment ofthe present disclosure.

FIG. 9 is a structural diagram illustrating an electronic deviceaccording to at least one embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to make the purpose, technical scheme and advantages of thepresent disclosure more apparent, example embodiments according to thepresent disclosure will be described in detail below with reference tothe accompanying drawings. Obviously, the examples described are onlypartial examples of the present disclosure, not the entirety of thepresent disclosure, and it is to be understood that the presentdisclosure is not limited by the example embodiments described herein.

In the present specification and drawings, steps and elements havingsubstantially the same or similar steps and elements are represented bythe same or similar drawings markings, and repetitive descriptions ofsuch steps and elements will be omitted. Also, in the description of thepresent disclosure, the terms “first,” “second,” etc. are used only todistinguish the described elements and are not to be understood asindicating or implying relative importance or order.

On one aspect, in the relevant quality inspection procedure of theproduct manufacturing, there are many unstable factors in manualinspection, which may lead to a decrease in the accuracy of the qualityinspection, thus causing potential problems in product quality. Onanother aspect, in the quality inspection procedure, all the data ismanually input, which is inefficient, and at the same time, informationobtained manually on an image of a product to be inspected within alimited time is relatively coarse, which brings inconvenience to thesubsequent search and analysis of defect causes. Based on all or part ofthe above reasons, the present disclosure provides the followingembodiments.

The products mentioned below include raw materials in the actualmanufacturing procedure, as well as semi-finished or finished productsafter each process (which is performed by devices used for productmanufacturing). For example, in the semiconductor industry, productsinclude glasses that have entered the production-line in the verybeginning, array substrates that have gone through the exposure process,screens that have gone through cell process, etc. Product images includeproduct images directly obtained by image acquisition devices (such ascameras, automated optical inspection (AOI) devices, etc.), as well asproduct images that each contain a defect content label (i.e., productimages that have gone through the identification of product defectcontent).

FIG. 1 is a schematic diagram illustrating an example scenario 100 forprocessing product manufacturing messages.

As shown in FIG. 1 , a plurality of products pass through a site 101 inturn in the scenario 100. The site 101 represents a place-point in thewhole production-line flow through which the products may pass.

The site 101 may be a physical device that completes a process instandardized production on a product-line, or a system comprised ofmultiple physical devices. For example, as for the photolithographyprocess of array substrates in the semiconductor industry, the site 101corresponding to this photolithography process may include a systemcomprised of a cleaning device, a pre-baking device, a cooling device, acoating device, an exposure device, a developing device, a post-bakingdevice, a cooling device, etc. The site 101 may also be a single device(an exposure device) corresponding to the exposure process or an AOIdevice corresponding to image detection. The site 101 may also be avirtual site in the product manufacturing procedure, which representsthe steps for processing products in a non-entity form. For example, thesite 101 may perform a procedure for defect detection (also referred toas inspection) of the products, which obtains and analyzes all of theprocedure information used to detect product defects, and thenidentifies the product defects. If the products enter the site 101, atrackin message is captured by the site 101. If the products leave thesite 101, a trackout message is captured by the site 101. In order toensure the product quality, product information/product data in thetrackin message and the trackout message need to satisfy therequirements of product manufacturing.

The site 101 may include a serving device (102) for productmanufacturing messages (hereinafter, referred to as productmanufacturing message serving device (102), a processing device (103)for product manufacturing messages (hereinafter, referred to as productmanufacturing message processing device (103), and an assistingdevice(s) (104) for product manufacturing (hereinafter, referred to asproduct manufacturing assisting device (104). The product manufacturingmessage serving device (102) may also be excluded from the site 101. Theproduct manufacturing message serving device (102), the productmanufacturing message processing device (103), and the productmanufacturing assisting device(s) (104) may be computing devices thatinclude processors and memories. These devices may be connected witheach other via a network. The above devices may be directly orindirectly communicated with each other. For example, these devices cansend and receive data and/or signals via a network. The network may bethe Internet of Things based on the Internet and/or telecommunicationnetwork, which may be a wired network or a wireless network. Forexample, the network may be an electronic network that can realize thefunction of information exchange, such as a local area network (LAN), ametropolitan area network (MAN), a wide area network (WAN), and acellular data communication network. Each device may use one or morecommunication protocols to communicate with each other, such as FTP,TCP/IP, HSMS, and Tibco.

In the present disclosure, the site 101 is mainly applied to thedetection, analysis, and processing of product defects. It should beunderstood by those skilled in the art that the site 101 may also beapplied to other procedures of product manufacturing.

The product manufacturing message serving device (102) may be configuredto capture all or part of the product manufacturing messages in theproduct manufacturing procedure (for example, the trackin message asabove), and broadcast or send these product manufacturing messages tothe product manufacturing message processing device (103). The productmanufacturing message processing device (103) may be configured toperform further processing on the product manufacturing messages, andsend a task message to the product manufacturing assisting device(s)(104) to perform detection and analysis of product defects. The productmanufacturing assisting device (104) may include one or more of thefollowing: a device for inspectors to inspect product defects, a devicefor detecting product defects using an AI defect identification model, adevice for deploying an AI defect identification model, a device fortraining an AI defect identification model, a device for alertingproduct defects, and the like. The product manufacturing assistingdevice (104) may return an analysis result to the product manufacturingmessage processing device (103) after completing the analysis anddetection of product defects.

FIG. 2 is a flowchart illustrating a method (200) for processing productmanufacturing messages according to at least one embodiment of thepresent disclosure.

The method (200) for processing product manufacturing messages mayinclude some or all of the operations shown in FIG. 2 (e.g., some or allof operation 210 to operation 230). Of course, the method (200) forprocessing product manufacturing messages may also include otheroperations not shown in FIG. 2 . The method (200) for processing productmanufacturing messages may also be performed by any other electronicdevice capable of communication and computing. Below, a productmanufacturing message processing device (103) is illustrated to performthe method 200, as an example.

See FIG. 2 . In operation 210, a plurality of product manufacturingmessages may be monitored by the product manufacturing messageprocessing device (103).

During intelligent product manufacturing, a large number of productmanufacturing messages are generated. These product manufacturingmessages can be used to indicate the manufacturing procedure of aproduct, or to indicate a possible defect of the product in themanufacturing procedure. The product manufacturing messages may includerecord information generated by any of the devices used for productmanufacturing through which the product passes. Through the productmanufacturing messages, it is known that the product has been processedby the product manufacturing device, and other related processingresults may also be known. For site 101, the product manufacturingmessages include a message for indicating that a product image isgenerated.

For example, during the detection procedure of screen defects in thesemiconductor industry, the product manufacturing device may be theabovementioned Automated Optical Inspection (AOI) device. The productmanufacturing device can be configured to perform an optical inspectionon the products and capture images of the products in the manufacturingprocedure, to determine the differences between the captured productimages and standard product images. Based on these differences, theproduct manufacturing device can determine the presence of productdefects in the products being detected. The product manufacturing devicemay also be other cameras or photographic cameras having an imageacquisition function. The product manufacturing device may send capturedproduct images and corresponding files to a product image database. Theproduct image database may be a distributed file system (DFS) or otherdata storage devices. The corresponding message indicating thegeneration of product images may include a message indicating that theproducts have entered a product manufacturing device (e.g., an AOIdevice) (e.g., a trackin message) and/or a message indicating that theproducts have left the product manufacturing device (e.g., an AOIdevice) (e.g., a trackout message), and may also include a messageindicating that the product manufacturing device (e.g., an AOI device)generates product image files or a message indicating that productimages are sent to the product image database. The product manufacturingdevice may also be other devices that can be used for productmanufacturing, without limitation of this disclosure.

The product manufacturing message serving device (102) may be configuredto capture all or part of the product manufacturing messages in theproduct manufacturing procedure, and broadcast or send these productmanufacturing messages to the product manufacturing message processingdevice (103). The product manufacturing device may also be configured tosend product manufacturing messages directly to the productmanufacturing message processing device (103). The product manufacturingmessages obtained by the product manufacturing message processing device(103) are the plurality of product manufacturing messages which aremonitored.

For example, the product manufacturing message serving device (102)includes a manufacturing execution system (MES), and may also include anexecutive information system (EIS). The product manufacturing messageserving device (102) may also be other devices used for monitoringproduct manufacturing, which is not limited in the present disclosure.Therefore, the product manufacturing messages can be generated by theproduct manufacturing device or captured by the product manufacturingmessage serving device (102). The product manufacturing messages mayinclude product manufacturing site information and/or productinformation. The product manufacturing site information includes theidentity of the site, the physical location of the site (e.g., thephysical location of the AOI device), the process-node information ofthe site in the product manufacturing procedures (e.g., defectidentification/detection in the exposure process, defectidentification/detection in the cleaning process, etc.), etc. Theinformation can be used to assist the product manufacturing messageserving device (102) to identify or position a specific site. Theproduct information may be product type, product name, product identity,product priority, etc. This information can be used to assist theproduct manufacturing message serving device (102) to identify orposition a specific product. It should be understood by those skilled inthe art that the contents of the product manufacturing messages and theproduct information are not limited to the above examples, as long asthe contents thereof are related to defect identification/detection inthe product manufacturing procedure.

The above-mentioned product manufacturing messages include at least oneLOT-products manufacturing message and at least one single-productmanufacturing message. For example, when screen products of the productproduction-line are inspected by the AOI device, a single-productmanufacturing message (for example, GlassTrackOut message) may be sentas a product manufacturing message from the AOI device to the productmanufacturing message serving device (102) after the inspection on onescreen (or large glass substrate screen, also referred to as Glass) iscompleted, and picture files (.jpg/.gls) may be sent from the AOI deviceto the product image database 203. A single-product manufacturingmessage may also be sent from the AOI device to all the activateddevices in the current factory. The AOI device may also be configured tosend a LOT-products manufacturing message by taking a LOT as a unit (1LOT contains 20 Glass, and each Glass is a single large glass substratescreen). For example, if the inspection of one LOT is completed, aLOT-products manufacturing message (e.g., LotTrackOut message), taken asone product manufacturing message, may be sent from the AOI device tothe product manufacturing message serving device (102) or any otherrelevant devices.

In industrial production, a plurality of products are combined into oneLOT, and the same LOT is subjected to the same processing process, so asto facilitate the recording and sorting of the product manufacturingmessages. One LOT-products manufacturing message refers to a collectionof product manufacturing messages of the plurality of products of thesame LOT, and one single-product manufacturing message refers to productmanufacturing messages of a single product (such as GLASS). A relativelylong cycle is needed to generate a LOT-products manufacturing messageand a relatively short cycle is needed to generate a single-productmanufacturing message. To improve the processing efficiency of productmanufacturing messages, optionally, in operation 210, the monitoring ofthe plurality of product manufacturing messages also includes:monitoring the LOT-products manufacturing messages by interrupt andmonitoring the single-product manufacturing messages by polling. Themonitoring the LOT-products manufacturing messages by interrupt refersto, after monitoring the first LOT-products manufacturing message,stopping monitoring until the next LOT-products manufacturing message isgenerated, and then continue monitoring again. The monitoring thesingle-product manufacturing messages by polling refers to continuouslymonitoring a device of generating the single-product manufacturingmessages at a preset frequency. In the product manufacturing procedure,the recording and delivery of product manufacturing messages is usuallyperformed in the unit of LOT-products (LOT), which can improve theprocessing efficiency of messages. However, during the task foranalyzing product defects, if only the LOT-products manufacturingmessages are monitored, the product manufacturing message processingdevice (103) and devices for the inspection and analysis of the productdefects are often in an idle state. Thus, the product manufacturingmessage processing device (103) may monitor the LOT-productsmanufacturing messages by interrupt, monitor the single-productmanufacturing messages by polling during interruption intervals, andprocess the single-product manufacturing messages in a timely manner.The single product corresponding to the single-product manufacturingmessages may or may not be one product of the products in the LOTcorresponding to the LOT-products manufacturing messages, which allowsthe processing of most of the single-product manufacturing messages forthe LOT to have been completed by the time the LOT-productsmanufacturing message for the LOT is received. After completing theprocessing of all product manufacturing messages for the LOT (e.g., oneLOT), the processing of product manufacturing messages for the next LOTis then performed, thereby improving the message processing efficiencyof the product manufacturing message processing device (103).

Optionally, in order to monitor the LOT-products manufacturing messagesby interrupt, the product manufacturing message processing device (103)may also register, with the product manufacturing message serving device(102), information about the LOT-products manufacturing messages itwishes to monitor. The details of the registration procedure will bedescribed in subsequent embodiments of the present disclosure. Thereby,a LOT-products manufacturing message is broadcast by the productmanufacturing message serving device (102) to the product manufacturingmessage processing device (103) after the manufacturing of the LOT ofproducts is completed. The product manufacturing message processingdevice (103) then receives the LOT-products manufacturing message, andwhen the LOT-products manufacturing message is received, performs aninterruption. During the interruption interval, the productmanufacturing message processing device (103) may begin monitoring forsingle-product manufacturing messages by polling. When a LOT-productsmanufacturing message is available again, the product manufacturingmessage serving device (102) broadcasts it to the product manufacturingmessage processing device (103).

In operation 220, a product defect analysis task queue may beestablished by the product manufacturing message processing device (103)based on the plurality of product manufacturing messages.

As described above, the site 101 may be taken as a detection site in theentire product manufacturing procedure. Products of differentproduction-lines of the factory may enter the site 101 in the detectionprocedure. At present, in the factory, due to various types of productsand complex processes, a wide variety of product manufacturing sites andcomplex product defects are present. In this case, the frequency and thequantity of the products flowing into (entering) the site 101 areuncertain, the quantity of products that have flew into (entered) thesite 101 and that need to be detected sometimes suddenly increases to alarge number, and sometimes is of a small number. Therefore, areasonable scheduling and distribution of detection tasks for variousproducts is required. In order to make a plurality of productmanufacturing assisting devices (104) perform the detection and analysisof the products orderly and efficiently product manufacturing assistingdevices (104), the product manufacturing message processing device (103)will establish a product defect analysis task queue based on thereceived product manufacturing messages to distribute tasks in the orderof the product defect analysis task queue.

In operation 230, product defect analysis tasks are distributed to theproduct manufacturing assisting devices based on the product defectanalysis task queue. The product defect analysis tasks include a task ofidentifying product defect content based on a defect identificationmodel.

Taking product defects as an example, the causes of product defects inthe manufacturing procedure vary, such as insufficient strength in thecleaning process, insufficient corrosion, excessive corrosion,inaccurate matching of raw materials, excessive micro-dust in thecleaning environment, insufficient exposure strength, excessive exposurestrength, and foreign matter doping, during the semiconductor productionprocedure. Therefore, further analysis of a product defect is requiredto obtain the product defect type, defect location (e.g., the circuitboard, the layer-level and the mask layer where the defect is located,the specific coordinate position on the board (e.g., the coordinates ofthe vertices of the peripheral rectangle, which can also be expressed asthe coordinates of a vertex plus the length and width), the relationshipof the defect to the shape of the background circuit model (e.g.,between two lines on a Gate Island, the number of Gate Islands coveredby the defect region, whether the defect falls entirely within a GateIsland, intersects the Gate Island, or is near a Gate Island, etc.), andthe defect size (e.g., the length of the defect or the area of thedefect region (e.g., pixel area).

Different product manufacturing assisting devices (104) may be requiredfor processing different product defect analysis tasks. The productmanufacturing assisting devices (104) includes: a first productmanufacturing assisting device (104-1) that configures and manages an AIdefect identification model, a second product manufacturing assistingdevice (104-2) that detects and analyzes the product defects in an AImanner, and a third product manufacturing assisting device (104-3) thatdetects the product defects based on manual intervention, etc. The firstproduct manufacturing assisting device (104-1) may be one or moredevices (e.g., a model management cluster) that sets parameters of theAI defect identification model and manages the AI training procedure ofthe other product manufacturing assisting devices for productmanufacturing. The second product manufacturing assisting device (104-2)may be one or more devices (e.g., a product defect analysis cluster)capable of performing inference and training tasks for AI defectidentification models and for scheduling and allocation of hardwareresources utilizing GPU computational resources. The third productmanufacturing assisting device (104-3) may be a terminal (e.g., aproduct manufacturing client device) that presents product defects to arelevant staff and allows him or her to make a judgment about theproduct defects. As an example, the product manufacturing client devicein a factory can display product defect images to the relevant staff andthe relevant staff then judge the product defects, set the relevantinformation, analyze the relevant data based on the defect images, orthe relevant staff may then take a defect judgment examination based onthe defect images.

The product defect analysis task also includes a task of identifyingproduct defect content based on the AI defect identification model. Theproduct defect content includes any one or more of: defect type, defectlocation, and defect size of products. The product defect analysis taskmay also include a training task of the AI defect identification model.The AI defect identification model includes one or more of: afeedforward neural network AI defect identification model, aconvolutional neural network model, a cyclic neural network model, or agenerative adversarial network model.

In embodiments of the present disclosure, the task of identifyingproduct defect content based on the AI defect identification model isimplemented as follows. Firstly, a product image is scaled to a fixedpixel size M×N (may also be not scaled), and then the M×N image is sentto a deep convolutional neural network (for example,VGG/Resnet/MobileNet, etc.). Secondly, feature maps of the entire M×Nimage are obtained after the M×N image has passed through multipleconvolutional layers, activation layers, and pooling layers. Thirdly,the feature maps are input into a region proposal network (ZF/SSD/RPN,etc.), and proposal regions are obtained by calculation. Fourthly,proposal feature maps of the proposal regions are obtained by performingoperations (such as convolution and pooling) on the proposal regions,and the proposal feature maps are sent to the subsequent fully-connectednetwork and a softmax network for classification (i.e, to classify theproposal into a defect type). The defect type with the largestprobability is determined as the final classification result, and thedefect type and the probability are recorded. In addition, thecoordinate and the size of the proposal region represent the positionand the size of the defect. The method of identifying the product defectcontent based on the AI defect identification model can adopt similarvariations of the above method or other methods known to those skilledin the art, which is not limited in the present disclosure.

The second product manufacturing assisting device (104-2) can be used toprocess a product defect analysis task of identifying product defectcontent from product images using the AI defect identification model.The second product manufacturing assisting device (104-2) may be one ormore devices capable of performing inference and training tasks of theAI defect identification model using GPU (Graphics Processing Unit)computing resources.

The AI defect identification model is primarily based on neuralnetworks. For example, an AI defect identification model may be based ona feedforward neural network, i.e., a feedforward neural network model(also referred to as feedforward network). The feedforward network canbe implemented as an acyclic graph, in which nodes are arranged inlayers. Generally, the feedforward network includes an input layer andan output layer, and the input layer and output layer are separated byat least one hidden layer. The hidden layer transforms the inputreceived by the input layer into a useful representation for generatingoutput in the output layer. Network nodes are fully connected to nodesin adjacent layers via edges, but there are no edges between nodes inthe same layer. Data received at the nodes of the input layer of thefeedforward network is propagated (namely “feedforwarded”) to nodes ofthe output layer through an activation function. The status of the nodesof each continuous layer in the network is calculated by the activationfunction based on coefficients (“weights”), and the coefficients arerespectively related to each of the edges that connect these layers. Theoutput of the AI defect identification model may adopt various forms,which is not limited in the present disclosure. The AI defectidentification model may also include other neural network models suchas a convolutional neural network (CNN) model, a recurrent neuralnetwork (RNN) model, or a generative adversarial network (GAN) model,but the present disclosure is not limited thereto. Other neural networkmodels that are commonly known by those skilled in the art may also beadopted as the defect identification model.

The second product manufacturing assisting device (104-2) may be furtherconfigured to train the neural network model, which mainly includes thefollowing steps: for example, selecting a network topology; using a setof training data that represents problems modeled by the networktopology; and adjusting the weights until the AI defect identificationmodel has the smallest error for all instances of the set of trainingdata. For example, in the supervised learning training procedure for aneural network, the output generated by the network in response to aninput representing an instance in the training data set is compared withthe labeled output “correct” of the instance; the error signal thatindicates the difference between the output and the labeled output iscalculated; and if the error signal is propagated backwards through thelayers of the network, the weights associated with the connections areadjusted to minimize the error. If the error of each output generatedfrom corresponding instance of the set of training data is minimized,the AI defect identification model is construed as “has been trained”.

The accuracy of the AI defect identification model can be greatlyaffected by the quality of the dataset used to train said algorithm (themodel). The training procedure can be computationally intensive, so itis beneficial to use GPUs to train many types of AI defectidentification models. The calculations performed in tuning thecoefficients in the neural network are naturally suited to parallelimplementations. Specifically, many machine learning algorithms andsoftware applications have been adapted to use parallel processinghardware within general-purpose graphics product manufacturing messageprocessing devices. It is efficient in processing the calculationsassociated with training deep neural networks. Thus, the use of a GPUcluster with multiple integrated GPUs can effectively increase thetraining and inference speed of AI defect identification models. Thesecond product manufacturing assisting device (104-2) can also scheduleand allocate the hardware resources.

Thereby, the product manufacturing message processing device (103) maydetermine whether an AI defect identification model is required fordetection according to a product type, whether relevant models need tobe trained, and distribute different product defect analysis tasks tothe first product manufacturing assisting device (104-1) to the thirdproduct manufacturing assisting device (104-3) based on thedetermination results. For example, a defect content identification taskbased on an AI defect identification model can be generated for a knownproduct type (e.g., a trained product). For unknown product types (e.g.,new untrained products), a defect content identification task based onmanual-intervention (e.g., by operators) identification can begenerated. In addition, for product images with product defects thatcannot be identified by the AI defect identification model (e.g., the AIidentification probability is below a preset threshold), the productmanufacturing message processing device (103) may also generate a defectcontent identification task based on the manual-interventionidentification. The product manufacturing message processing device(103) can also classify the tasks distributed to the first productmanufacturing assisting device (104-1), the second product manufacturingassisting device (104-2), and the third product manufacturing assistingdevice (104-3) based on the quantity of tasks in the product defectanalysis task queue, allowing computer resources and human resources tobe used more efficiently.

The method (200) for processing product manufacturing messages accordingto at least one embodiment of the present disclosure can improve theefficiency of the processing of product manufacturing messagesthroughout the whole product manufacturing procedure, so that eachdevice involved in the detection and analysis of product defectsoperates efficiently, facilitating subsequent finding and analysis ofcauses of the product defects, and improving the efficiency of productmanufacturing.

FIG. 3 is another flowchart illustrating the method (200) for processingproduct manufacturing messages according to at least one embodiment ofthe present disclosure, which illustrates the procedure of obtainingproduct manufacturing messages in the method (200) for processingproduct manufacturing messages, e.g., some or all sub-operations of theoperations 210 described above.

In sub-operation 211, a registration information is sent from theproduct manufacturing message processing device (103) to the productmanufacturing message serving device (102). The registration informationincludes product manufacturing site information and/or first productinformation.

Through the registration information, the product manufacturing messageserving device (102) can be aware of what product manufacturing messagesrelated to the registration information the product manufacturingmessage processing device (103) would like to know. The productmanufacturing messages related to the registration information includeproduct manufacturing site information or first product information.Thus, when the product manufacturing messages related to theregistration information have been collected, the product manufacturingmessage serving device (102) can preferentially broadcast the productmanufacturing messages. Thus, the product manufacturing messageprocessing device (103) can monitor the product manufacturing messagesby interrupt.

The product manufacturing site information includes one or more of: theidentity of the site, the physical position of the site (for example,the physical position of the AOI device), the process-node informationof the site in the product manufacturing procedure (for example, defectidentification/detection in the exposure process, defectidentification/detection in the cleaning process, etc.), etc. Thisinformation can be used to assist the product manufacturing messageserving device (102) to identify or locate a specific site. The firstproduct information may be one or more of: product type, product name,product identification, product priority, etc., which can be used toassist the product manufacturing message serving device (102) identifyor locate to a specific product. It should be understood by thoseskilled in the art that the contents of the product manufacturing siteinformation and the first product information are not limited to theabove example, as long as the contents are related to defectidentification/detection in the product manufacturing procedure.

In sub-operation 212, based on the registration information, a firstproduct manufacturing message sent from the product manufacturingmessage serving device (102) is monitored by the product manufacturingmessage processing device (103), in which the first productmanufacturing message includes product manufacturing messages related tothe registration information. The first product manufacturing messagemay be a LOT-products manufacturing message or a single-productmanufacturing message, as long as the first product manufacturingmessage is related to the product manufacturing site information or thefirst product information in the registration information. If the firstproduct manufacturing message is related to the product manufacturingsite information, the first product manufacturing message may includesite variation information, site status information, and the likeidentified by the product manufacturing message serving device (102). Ifthe first product manufacturing message is related to the first product(e.g., the first product information), the first product manufacturingmessage may include the address where the product images of the productare stored, the number of product images captured with respect to theproduct, the manufacturing procedure of the product, etc.

In sub-operation 213, a second product manufacturing message sent fromthe product manufacturing message serving device (102) is monitored bythe product manufacturing message processing device (103), in which thesecond product manufacturing message is not related to the registrationinformation. The product manufacturing message serving device (102) canuse the same port to broadcast the first product manufacturing messageand the second product manufacturing message, and may also use differentports to broadcast the first product manufacturing message and thesecond product manufacturing message, which is not limited in thepresent disclosure. Optionally, the second product manufacturing messagemay include information that is not related to the content of theregistration information, for example, the temperature, the humidity,and the like of the current factory environment. Of course, the secondproduct manufacturing message may be a LOT-products manufacturingmessage and may also be a single product manufacturing message.

In sub-operation 214, it is determined by the product manufacturingmessage processing device (103) whether the list of productmanufacturing keywords includes a product manufacturing keyword in thesecond product manufacturing message.

In the case where the product manufacturing keyword in the secondproduct manufacturing message is included in the list of productmanufacturing keywords, in sub-operation 215, the product manufacturingmessage processing device (103) reserves the second productmanufacturing message.

In the case where the product manufacturing keyword in the secondproduct manufacturing message is not included in the list of productmanufacturing keywords, in sub-operation 216, the product manufacturingmessage processing device (103) discards the second productmanufacturing message.

The registration of the product manufacturing message processing device(103) to the product manufacturing message serving device (102) mayrequire more procedures, such as configuration and verification byrelevant staff. However, in the factory, products may be continuouslyadjusted and updated. There may be cases where information about a newproduct needs to be obtained in time but the product manufacturingmessage processing device (103) cannot register to the productmanufacturing message serving device (102) timely. Thus, the productmanufacturing message processing device (103) may also monitor messageswhich are not related to the registration information.

For example, the messages which are not related to the registrationinformation may be monitored by polling. Product manufacturing keywords,which are not related to the registration information but related to theidentification, detection and analysis of product defects, are stored inthe list of product manufacturing keywords. When the second productmanufacturing message is received, the product manufacturing messageprocessing device (103) can analyze fields in the second productmanufacturing message, compare these fields with the list of productmanufacturing keywords, and reserve the second product manufacturingmessage containing any product manufacturing keywords in the list. Forexample, assuming that relevant staff finds that the product defects inthe screens produced in the last few LOTs are likely to be caused byexcessive environmental humidity, the relevant staff can add the keyword“environmental humidity” in the list of product manufacturing keywords.Subsequently, when the messages related to environmental humidity arereceived, the product manufacturing message processing device (103) canreserve such messages so as to facilitate the relevant staff to performthe analysis. When the influence factors such as environmental humidityare excluded, the relevant staff can remove the keywords related toenvironmental humidity in the list of product manufacturing keywords, soas to reduce redundant information stored in the product manufacturingmessage processing device (103) and improve the message processingefficiency of the product manufacturing message processing device (103).In the whole procedure of acquiring environmental humidity information,the product manufacturing message processing device (103) does not needto register to the product manufacturing message serving device (102),thereby reducing the procedures of processing the product manufacturingmessages.

By adoption of this mechanism, the product manufacturing messageprocessing device (103) can preferentially monitor registeredinformation, and meanwhile, filter non-registered information accordingto the product manufacturing keywords. The product manufacturing messageprocessing device (103) not only monitors the registered information butalso can monitor messages sent by all the product manufacturing messageserving devices (102), thereby improving the expandability of thesystem.

In the procedure of monitoring and broadcasting the productmanufacturing messages, the factory manufacturing system has a highrequirement for the processing of the product manufacturing messages.Optionally, to avoid message loss, the product manufacturing messageprocessing device (103) may also have the functions of message cache andmessage queue. The product manufacturing message processing device (103)can cache the product manufacturing messages in RabbitMQ for messagequeue management to avoid the problem of message loss caused by delay orother abnormities.

FIG. 4 is a schematic diagram illustrating the method (200) forprocessing product manufacturing messages according to at least oneembodiment of the present disclosure. FIG. 4 illustrates the procedureof establishing a product defect analysis task queue in the method (200)for processing product manufacturing messages.

As shown in FIG. 4 , as the frequency and the quantity of the productsentering the site 101 are uncertain, the quantity of products that havebeen entered the site 101 and that need to be detected sometimessuddenly increases to a large number, and sometimes is of a smallnumber. The quantity of tasks flowing into the site over time can beshown in a flowing-in task quantity graph.

If tasks flowing into the site are not scheduled, the productmanufacturing assisting device (104) to perform the detection andanalysis will sometimes be over-stressed and sometimes relatively idle,so the processing efficiency of the product defect analysis tasks can below. The product defect analysis task queue is disposed in the productmanufacturing message processing device for product manufacturingmessages and provides a time buffer between the production of theproduct images by the image acquisition device and the performing of theproduct defect analysis tasks by the product manufacturing assistingdevice for product manufacturing. The product defect analysis tasks inthe product defect analysis task queue include product defect analysistasks to be performed. The product manufacturing message processingdevice for product manufacturing messages can control the speed ofdistributing tasks in the product defect analysis task queue accordingto the load status of the product manufacturing assisting device forproduct manufacturing. If the quantity of the product manufacturingmessages is greater than a preset message threshold, and/or the quantityof tasks in the product defect analysis task queue is greater than apreset task threshold, and/or the load of the product manufacturingassisting device is greater than a preset load threshold, the speed ofdistributing tasks is reduced. If the quantity of the productmanufacturing messages is less than the preset message threshold, and/orthe quantity of tasks in the product defect analysis task queue is lessthan the preset task threshold, and/or the load of product manufacturingassisting device for product manufacturing is less than the preset loadthreshold, the speed of distributing tasks is increased.

Optionally, the product manufacturing message processing device (103)establishes the product defect analysis task queue based on theplurality of product manufacturing messages also includes: sorting theproduct defect analysis tasks based on any one or more of: the order inwhich the product manufacturing messages are received, priorities of theproducts, and a product scheduling plan to establish the product defectanalysis task queue.

In the cases, the sorting of the product defect analysis tasks based onthe order in which the product manufacturing messages are received is tofacilitate the scheduling of tasks according to time sequence. That isto say, according to the sequence of the monitored product manufacturingmessages entering the queue to be processed, the corresponding hardwareresource match is performed. As shown in FIG. 4 , assuming that 9product manufacturing messages are received in turn and 9 product defectanalysis tasks Job1 to Job9 are established on this basis, and then the9 tasks can be arranged according to the order in which the 9 tasks arereceived. The manner of sequential scheduling is simple to set, and canbe better matched with the production plan in terms of time.

The sorting of the product defect analysis tasks based on the prioritiesof the products includes scheduling according to the set productpriorities, that is, sorting according to the priorities of the productsin the production plan. The entire message queue for the scheduling isdynamically changed, and products with high priorities may be insertedinto front positions of the product defect analysis task queue. As shownin FIG. 4 , assuming that the priorities of Job2 and Job3 are higherthan those of Job1, then, Job2 and Job3 may be arranged before Job1 forprocessing. The scheduling based on priority can ensure the detectiontasks of the products with high priority are successfully completedfirst.

The sorting of the product defect analysis tasks based on the productscheduling plan may allow scheduling staff to be able to specify acorresponding scheduling plan, and meanwhile, support temporaryinsertion and adjustment of the scheduling plan. For example, as shownin FIG. 4 , Job9 is temporarily inserted between Job2 and Job3. Thus,the scheduling staff can focus on monitoring and verifying certainproducts or defects, and at the same time, can integrate otherinformation on the production-line to temporarily intervene in the jobsof the entire production-line.

Some embodiments of the present disclosure can combine any one or moreof the above three methods to sort the product defect analysis tasks toestablish the product defect analysis task queue. As shown in FIG. 4 ,the quantity of tasks flowing out from the site remained more or lesseven over time after the scheduling (as shown by a flowing-out taskquantity graph). The embodiments of the present disclosure can performreasonable scheduling and allocation on the detection and/or analysistasks of various products, and then perform reasonable allocation oncomputing resources and tasks, so as to satisfy the needs of actualproduction at maximum efficiency.

FIG. 5 is another flowchart illustrating the method (200) for processingproduct manufacturing messages according to at least one embodiment ofthe present disclosure, which illustrates an example of operation 220.

Referring to FIG. 5 , operation 220 may include all or a part ofsub-operations 221 through 227.

In sub-operation 221, a plurality of product images are obtained by theproduct manufacturing message processing device (103) based on theplurality of product manufacturing messages. Since an AOI devicecaptures a large number of high-resolution product images during productdefect detection (an AOI device may take a plurality of images of asingle large glass substrate screen), the AOI device sends these imagesto a product image database (such as a DFS system) for storage. Theproduct manufacturing messages sent by the AOI device includes fieldsindicating specific locations in the product image database where theimages are to be stored, so that the product manufacturing messageprocessing device (103) can retrieve these product images from theproduct image database.

In sub-operation 222, product defects are obtained by the productmanufacturing message processing device (103) from the plurality ofproduct images. The product manufacturing message processing device(103) will perform a preliminary analysis of these product images toobtain the product defects. For example, the product manufacturingmessage processing device (103) may locate locations of the productdefects, and indicate a quantity of product defects, and the like. Insome embodiments, the images to be analyzed are encoded, the encodedimages are then detected, to identify anomalous portions that do notconform to particular rules, in order that the anomalous portionstherein are eliminated and corrected. The images with anomalous portionseliminated and corrected are decoded and the decoded images are comparedto the images to be labeled to obtain the defect location, defect size,and quantity of defects, etc.

If an AI defect identification model corresponding to product defects ispresent, the product manufacturing message processing device (103) maygenerate a product defect analysis task for identifying the productdefects or inferring the causes of product defects (identifying productdefects includes identifying information about the defect type, defectlocation, etc. of the product defects). If the AI defect identificationmodel corresponding to the product defects is not present, the productmanufacturing message processing device (103) may generate a productdefect analysis task that trains the AI defect identification model. Theproduct manufacturing message processing device (103) may send a queryto the model management cluster described above to inquire whether an AIdefect identification model exists. The product manufacturing messageprocessing device (103) may also query its internally stored list of AIdefect identification models to inquire whether the AI defectidentification model is present. The present disclosure does not limitthe manner in which the product manufacturing message processing device(103) inquires whether an AI defect identification model is present.

Due to the wide variety of product defects and their varyingdistributions, the product manufacturing message processing device (103)pre-processes these product images in order to facilitate the productmanufacturing assisting device (104) to obtain better analysis resultsor better AI defect identification models.

For example, the distribution of the quantity of product defects in alarge glass substrate screen may be uneven. For example, the quantity ofdefects located in the middle of the glass substrate may far exceed thequantity of defects at the edges of the glass substrate. Ifpre-processing is not performed, it may lead to excessive deviation ofdata in the training set of the AI defect identification model, which inturn may lead to poor effect of identification using the AI model.Product defects may also be inconspicuous, e.g., on a large glasssubstrate screen, defects may only be present in a very small region ofpixel blocks. If such an image is fed directly into the AI defectidentification model, it may also result in poor effect. Thecharacteristics between multiple product defects may also be obscure,and product defects from different causes may have similarcharacteristics, which may also lead to poor effect of identification.To identify product defects, a large number of AI defect identificationmodels may be trained in site 101. However, when these models aretrained, the contrast between positive and negative samples in theproduct images may not be obvious, resulting in poor effect ofidentification using the AI defect identification models at the trainingsite. In summary, if product images are not pre-processed, it may resultin increased difficulty and poor effect of identification of the productmanufacturing assisting device (104).

Thus, the product manufacturing message processing device (103) may makethe following determinations and perform the following processes on theproduct images to improve the identification effect of the AI defectidentification model. Optionally, operation 222 also includes any one ormore of sub-operation 223 to 226.

In sub-operation 223, in the case of an uneven distribution of thequantity of product defects, the product manufacturing messageprocessing device (103) performs any one or more of: rotation, scaling,color transformation, and interception on a product image. Often, dataskewness, variance, and the like can be used to determine whetherproduct defects are evenly distributed. For example, if the productmanufacturing message processing device (103) determines that thequantity of product defects in the product image has a data skewnessgreater than a predetermined data skewness, the quantity of productdefects may be determined to have an uneven distribution. To identifywhether there is an uneven distribution of the quantity of productdefects, the product manufacturing message processing device (103) mayperiodically perform sample counts according to product and site to seechanges in the distribution. The product manufacturing messageprocessing device (103) can expand product images that have a smallquantity of defects but have high process requirements. For example, theproduct manufacturing message processing device (103) may perform thefollowing processing on these product images: rotation, scaling, colortransformation, interception, etc. These processing means may expand thequantity of relevant samples, thereby allowing the AI defectidentification model to more effectively identify product defects.

In sub-operation 224, in the case where the image region where a productdefect is located is less than a first predetermined threshold, theimage region where the product defect is located is magnified by theproduct manufacturing message processing device (103). The firstpredetermined threshold can be the maximum area of the product defectregion, the maximum ratio of the product defect region to the imageregion, etc. For large product images with small product defect regions,the product manufacturing message processing device (103) can performsimple image processing. For example, the product manufacturing messageprocessing device (103) may initially screen out similar images andperform image segmentation. The product manufacturing message processingdevice (103) may then process the segmented image, e.g., magnify theregion where the defect is located.

In sub-operation 225, in the case where the similarity between any twoproduct defects is greater than a second predetermined threshold,product images with said any two product defects are merged by theproduct manufacturing message processing device (103). The secondpredetermined threshold may be the maximum similarity of any two productdefects. The similarity of the product defect images may becharacterized by cosine similarity, Euclidean distance, or Manhattandistance, which is not limited in this disclosure. In a case where thefeatures between product defects are not obvious, the productmanufacturing message processing device (103) can merge similar productdefects for processing during coarse classification and subsequentlyperform a refine the processing. In the phase of training the AI defectidentification model, the product manufacturing message processingdevice (103) may merge different product defects with similar featuresfor processing in order to increase the overall number of samples. Thesample set of an increased number will be used as samples for trainingthe AI defect identification model.

In sub-operation 226, in the case where the similarity between positivesamples and negative samples in the product image is greater than athird predetermined threshold, a frequency domain image of the productimage is obtained by the product manufacturing message processing device(103), and the positive or negative samples are adjusted based on thefrequency domain image. The third predetermined threshold may be aminimum similarity between the positive samples and the negativesamples. In response to the above-mentioned case where the positive andnegative samples are too similar, the product manufacturing messageprocessing device (103) may adjust the images of the positive samplesand the negative samples by means of image processing to decrease thesimilarity between them. The product manufacturing message processingdevice (103) may obtain the frequency domain image via a Fouriertransform or wavelet. The statistical characteristics of positive andnegative samples in the product image is then calculated and analyzedbased on the obtained frequency domain image. Based on these statisticalcharacteristics, the product manufacturing message processing device(103) may initially process the frequency domain image (e.g., pass itthrough any one or more of a high-pass filter, a low-pass filter, or aband-pass filter to enhance the frequency domain image) and then convertthe frequency domain image to a time domain image by inversetransformation. Afterwards, the inspector or reviewer can review labelsof positive and negative samples and determine if the converted image isdistorted. This greatly reduces the manual workload and increases thespeed of sample collection.

In sub-operation 227, product defect analysis tasks are generated by theproduct manufacturing message processing device (103) based on theproduct defects.

The product manufacturing message processing device (103), afterpre-processing the product defects, can generate a product defectanalysis task based on analysis results of the product defects and theadjusted product image.

The product defect analysis tasks may include a product defect analysistask that trains an AI defect identification model, a product defectanalysis task that performs AI inference on the causes of productdefects, a product defect analysis task that performs AI inference andidentification of product defect content (i.e., a task of identifyingproduct defect content based on an AI defect identification model), aproduct defect analysis task that sets up an AI defect identificationmodel, a product defect analysis task that allows relevant staff toreview samples of product defect models (AI defect identificationmodels), and a product defect analysis task that allows relevant staffto make judgments about product defects, or the like. There is nolimitation to the types of product defect analysis tasks in thisdisclosure, as long as it is associated with product defect analysis.

After the product defect analysis tasks are generated, the productmanufacturing message processing device (103) will establish a productdefect analysis task queue and distribute the product defect analysistasks to the product manufacturing assisting devices (104) based on theproduct defect analysis task queue.

Optionally, the product manufacturing message processing device (103)may also obtain product type and product defect analysis task type fromthe plurality of product manufacturing messages, and, based on theproduct type and the product defect analysis task type, generate aproduct defect analysis request message and distribute the productdefect analysis tasks by sending the product defect analysis requestmessage. The product defect analysis request message may also include aproduct level (e.g., task priority).

The product manufacturing messages sent from the AOI device and theproduct manufacturing message serving device (102) may includestatistical information for an entire LOT of products (e.g., one LOT ofproducts) or all manufacturing messages for a single product. There area large number of fields in these messages that are not relevant toproduct defect identification. Thus, the product manufacturing messageprocessing device (103) is required to parse the product manufacturingmessages, while in accordance with the requirements of the productmanufacturing assisting device (104), encapsulating them into productdefect analysis request messages that the product manufacturingassisting device (104) can identify. For example, during productmanufacturing message analysis, the product manufacturing messageprocessing device (103) may obtain the quantity of AOI color images, theproduct type of the AOI color images, the size of the AOI color images,and the like, for a LOT of products. The AOI color images may be imagesof the semiconductor screens taken at any step of being substrates,deposition, etching, and box. In generating the product defect analysisrequest message, the product manufacturing message processing device(103) needs to determine the product type to be analyzed in the productdefect analysis task, and the type of product defect analysis task to beperformed (e.g., inference, training, status query, etc.). Afterwards,the product manufacturing message processing device (103) may alsoperform message format verification on the product defect analysisrequest message, and then distribute the product defect analysis tasksto the product manufacturing assisting devices (104) by sending theproduct defect analysis request message after determining that theformat of the product defect analysis request message is qualified. Thetype of product defect analysis task is used to indicate differentproduct defect analysis tasks.

FIG. 6 is another schematic diagram illustrating the method (200) forprocessing product manufacturing messages according to at least oneembodiment of the present disclosure.

See FIG. 6 , optionally, the distribution of the product defect analysistasks by the product manufacturing message processing device (103) tothe product manufacturing assisting devices based on the product defectanalysis task queue also includes sub-operations 231 to sub-operations235.

In sub-operation 231, it is determined by the product manufacturingmessage processing device (103) whether an AI defect identificationmodel corresponding to the product type is present.

The determination by the product manufacturing message processing device(103) that whether an AI defect identification model corresponding tothe product type is present includes: determining based on the producttype in the product manufacturing messages.

If the product type indicates a known product (i.e., a trained product)and it is determined that an AI defect identification modelcorresponding to the product type is present, the product defectanalysis tasks may be performed based on the AI defect identificationmodel corresponding to the product type.

If the product type indicates an unknown product (i.e., an untrainedproduct), it is determined that the AI defect identification modelcorresponding to the product type is not present.

If the product type is known product type, but the product defectcontent cannot be identified based on the AI defect identification modelcorresponding to the product type (for example, the identification scoreis lower than a preset threshold), it may also be determined that the AIdefect identification model corresponding to the product type is notpresent. Optionally, the identification score represents the probabilitythat the AI defect identification model identifies the product defectsof the defect type.

If the performances of the AI defect identification model areinsufficient to satisfy the product defect analysis tasks (e.g., theaccuracy, the precision and the recall are lower than a presetthreshold) corresponding to the product type, it may also be determinedthat the AI defect identification model corresponding to the producttype is not present.

The product manufacturing message processing device (103) may alsodetermine whether the AI defect identification model corresponding tothe product type is present by other means such as manual detection anddefect identification model prejudgment. There is no limitation to howthe product manufacturing message processing device (103) determineswhether the AI defect identification model corresponding to the producttype is present in the present disclosure.

In the case where the AI defect identification model corresponding tothe product type is not present, in sub-operation 232, the productmanufacturing message processing device (103) sends a first productdefect analysis request message to the first product manufacturingassisting device (104-1). The first product defect analysis requestmessage includes the product type, the storage address of productimages, the quantity of product images, and the task identity fortraining an AI defect identification model (the task identity indicatestraining an defect identification AI model corresponding to the productsof the product type by utilizing the storage address of the productimages and the quantity of the product images). The first product defectanalysis request message corresponds to the distribution of the modeltraining task in the product defect analysis tasks. The first productmanufacturing assisting device (104-1) may be a model managementcluster, which may perform functions such as configuring or managing,for example, the training of the AI defect identification model. Forexample, the first product manufacturing assisting device (104-1) canrealize the function of managing the training task of the AI defectidentification model.

Due to the large number of products and sites, the use of general AIdefect identification models will not solve all problems, so there is aneed to provide various AI defect identification models to makejudgments about different types of products. There may also be a need tofine-tune the product models during the production in a factory. Forexample, when the manufacturing procedure in the production-line of thefactory changes, or after a change in production equipment, theresulting AOI images will change. Such changes will result in a decreasein the accuracy of the AI defect identification models corresponding tothe factory production line, and the models will need to be retrained.

The first product manufacturing assisting device (104-1) may managethese AI defect identification models based on site, product, time node,etc., and configure training and testing of new models, etc. Since eachAI defect identification model needs to be trained, verified, tested,and evaluated before it can be used, in the case where the computationalresources for training the AI defect identification models are limited,the first product manufacturing assisting device (104-1) can reasonablyschedule and distribute training tasks of the AI defect identificationmodels according to the priority, the training quantity, and the stateof the hardware resources.

The first product manufacturing assisting device (104-1) may alsorealize the function of visualizing the training of the AI defectidentification models, so that the relevant staff can observe the statusof the training of the AI defect identification models in real-time (forexample, whether the training of an AI defect identification modelconverges), and determine whether to stop or adjust the trainingprocedure. With the continuous enrichment of data sets and thecontinuous optimization of training parameters, as for the same productor product defect detection, if the newly trained defect identificationmodel has a better effect than the old defect identification model, thefirst product manufacturing assisting device (104-1) may also update theAI defect identification model in a defect identification model library.The first product manufacturing assisting device (104-1) can also launcha training task and evaluate the AI defect identification model in realtime according to procedure information (for example, Loss, verificationaccuracy, etc.) that is feedback in the model training procedure, todetermine if the model training procedure is working properly. The firstproduct manufacturing assisting device (104-1) may also be configured toadd, delete, modify and edit training sets, verification sets, and testsets required for training. The first product manufacturing assistingdevice (104-1) may also be configured to deploy the AI defectidentification model according to product requirements and performevaluation and testing, etc. before deployment. The foregoingdescription of the functions of the first product manufacturingassisting device (104-1) is merely exemplary, and one of skill in theart should understand that the first product manufacturing assistingdevice (104-1) may also perform many more functions not mentioned, andthe present disclosure is not limited thereto.

After the first product manufacturing assisting device (104-1) completesthe product defect analysis task (e.g., determining the AI defectidentification model corresponding to the product type), the firstproduct manufacturing assisting device (104-1) may send a first productdefect analysis response message to the product manufacturing messageprocessing device (103). In sub-operation 233, the first product defectanalysis response message sent by the first product manufacturingassistant (104-1) is received by the product manufacturing messageprocessing device (103). The first product defect analysis responsemessage includes one or more of the following: the identification,accuracy, and recall of the AI defect identification model. Among otherthings, the AI defect identification model is determined based on theproduct type, the storage address of the product images, and thequantity of product images in the first product defect analysis requestmessage.

In a case where the AI defect identification model corresponding to theproduct type is present, in sub-operation 234, a second product defectanalysis request message is sent by the product manufacturing messageprocessing device (103) to the second product manufacturing assistingdevice. The second product defect analysis request message includes theproduct type, the storage address of the product images, and thequantity of product images. The second product defect analysis requestmessage corresponds to the task of identifying product defect contentbased on the AI defect identification model among the product defectanalysis tasks.

The second product manufacturing assisting device (104-2) may be a GPUcluster, which may realize, for example, functions such as inferring andtraining tasks by AI defect identification models using GPU computingresources, scheduling and allocation of hardware resources, and thelike. For example, the second product manufacturing assisting device(104-2) may implement the functionality of loading the AI defectidentification models. As there are a large number of products and sitesand the loading time of the AI defect identification models is long, thesecond product manufacturing assisting device (104-2) may load the AIdefect identification models in advance by default. After the settingsare changed, the second product manufacturing assisting device (104-2)may also utilize an independent background server to complete theloading of the models, so as to avoid consuming a large amount of timerequired for loading the models when the product changes, therebyimproving the overall efficiency. As the production-line may generate alarge number of images every day for the detection of the defectidentification model algorithm, the second product manufacturingassisting device (104-2) can also perform reasonable scheduling anddistribution of GPU resources, thereby improving the use efficiency ofhardware resources. The second product manufacturing assisting device(104-2) may also test the AI defect identification models to determinehow well the AI defect identification models identify the defects. Theforegoing description of the functions of the second productmanufacturing assisting device (104-2) is only an example, and one ofskill in the art should understand that the second product manufacturingassisting device (104-2) may also perform many more functions notmentioned, and the present disclosure is not limited thereto.

After the second product manufacturing assisting device (104-2)completes the product defect analysis task (e.g., inferring andanalyzing the product defects), the second product manufacturingassisting device (104-2) may send a second product defect analysisresponse message to the product manufacturing message processing device(103). In sub-operation 235, the second product defect analysis responsemessage sent by the second product manufacturing assistant is receivedby the product manufacturing message processing device (103). The secondproduct defect analysis response message includes one or more of thefollowing: product image identification, product defect location,product defect identity, and repair identity. The product defectlocation, product defect identity, and repair identity are determinedbased on the product type, the storage address of the product images,and the quantity of the product images in the second product defectanalysis request.

Optionally, the product manufacturing message processing device (103)may also monitor one or more of the accuracy, precision, recall,F-score, and speed, with which the product manufacturing assistingdevice (104) processes the product defect analysis task. For example,one or more of accuracy, precision, recall, F-score, and speed, withwhich the product manufacturing assisting device (104) processes a taskof identifying product defect content based on an AI defectidentification model is monitored. The speed may refer to the speed atwhich the second product manufacturing assisting device (104-2)processes the product images (which may be measured in terms of thequantity of product images/sec, etc.), or the speed at which a single AIdefect identification model is trained (which may be measured in termsof the quantity of trained models/hour, etc.).

In the manufacturing procedure of the factory, the backgrounds of theproduct circuits are complex and diverse. Many product defects can beeasily confused, and the production plan of the products may also beadjusted according to product orders. These factors bring greatchallenges to model performances. Changes in products, changes inprocess, or adjustments of the product images may cause the performancesof the AI defect identification models to deteriorate. The productmanufacturing message processing device (103) may monitor theperformances of the AI defect identification models at a fixed period orin real-time. If the performances of any product defect identificationmodel cannot satisfy production requirements, adjustment is made inreal-time. For example, the product manufacturing message processingdevice (103) may alert in time if there is a problem, and deploy andconfirm a new defect identification model.

Optionally, the product manufacturing message processing device (103)may also monitor the inference performances of the product defectidentification models. The inference performance is one or more of:accuracy, precision, recall, FScore, and speed, with which the firstproduct defect identification task is processed based on the productdefect identification model. Among which, the speed may indicate thespeed at which the second product manufacturing assisting device (104-2)identifies the product images (which may be measured in terms of thequantity of product images/sec, etc.). The accuracy can be calculated bythe following formula (1). The precision can be calculated by thefollowing formula (2). The recall can be calculated by the followingformula (3). Subsequently, the FScore is calculated by the followingformula (4).

$\begin{matrix}{{{Accuracy} = \frac{\left( {{TP} + {TN}} \right)}{\left( {{TP} + {FP} + {TN} + {FN}} \right)}};} & (1)\end{matrix}$ $\begin{matrix}{{{Precision} = \frac{TP}{\left( {{TP} + {FP}} \right)}};} & (2)\end{matrix}$ $\begin{matrix}{{{Recall} = \frac{TP}{\left( {{TP} + {FN}} \right)}};} & (3)\end{matrix}$ $\begin{matrix}{{FScore} = {\frac{\left\lbrack {\left( {1 + \beta^{2}} \right) \cdot {Precision} \cdot {Recall}} \right\rbrack}{\left( {{\beta^{2} \cdot {Precision}} + {Recall}} \right)}.}} & (4)\end{matrix}$

Wherein, True-Positive (TP) indicates that the calculation result ispositive and the actual result is also positive, that is, thecalculation result obtained after the inference calculation by the AIdefect identification model is A (herein, A represents one result), andthe actual result is also A. At this point, the calculation result isconsistent with the actual result.

False-Positive (FP) indicates that the calculation result is positiveand the actual result is negative, that is, the calculation resultobtained after the inference calculation by the AI defect identificationmodel is A but the actual result is not A. At this point, thecalculation result is inconsistent with the actual result.

False-Negative (FN) indicates that the calculation result is negativeand the actual result is positive, that is, the calculation resultobtained after the inference calculation by the AI defect identificationmodel is not A but the actual result is A. At this point, thecalculation result is inconsistent with the actual result.

True-Negative (TN) indicates that the calculation result is negative andthe actual result is also negative, that is, the calculation resultobtained after the inference calculation by the AI defect identificationmodel is not A and the actual result is also not A. At this point, thecalculation result is consistent with the actual result.

In the manufacturing procedure of the factory, the backgrounds of theproduct circuits are complex and diverse. Many product defects can beeasily confused, and the production plan of the products may also beadjusted according to product orders. These factors bring greatchallenges to model performances. Changes in products, changes inprocess, or adjustments of the product images may cause the performancesof the AI defect identification models to deteriorate. The productmanufacturing message processing device (103) may monitor theperformances of the AI defect identification models at a fixed period orin real-time. If the performances of any product defect identificationmodel cannot satisfy production requirements, adjustment is made inreal-time. For example, the product manufacturing message processingdevice (103) may alert in time if there is a problem, and deploy andconfirm a new defect identification model.

The product manufacturing message processing device (103) may monitor,according to the following three methods, the accuracy, the precision,the recall, the FScore, and the like, with which the productmanufacturing assisting device (104) processes the product defectidentification tasks. It should be understood by those skilled in theart that the following three methods are only examples, and the productmanufacturing message processing device (103) may also monitor theproduct defect analysis tasks according to other methods, as long as anyof the accuracy, the precision, the recall or the FScore in processingthe product defect analysis tasks can be obtained.

First Method: Monitoring Through Standard Data Samples

The standard data samples may be pre-stored in the product manufacturingmessage processing device (103). The standard data samples may beexamined and verified by senior factory operators and senior inspectorsto determine that the selection range (for example, the defect type, thedefect quantity, the defect distribution, etc.) of the samples is keptconsistent with that of training samples of the AI defect identificationmodel. Subsequently, the product manufacturing message processing device(103) may compare the inference result through the AI defectidentification model with corresponding manually set standard result,and then perform statistics on the accuracy, the precision, the recall,or the FScore and processing speed of the AI defect identificationmodel. Meanwhile, the product manufacturing message processing device(103) may also be configured to update the standard data samplesaccording to the time period, production condition, manual adjustment,etc.

Second Method: Monitoring Through an Online Sampling Model

In the manufacturing procedure of the factory, a plurality of (forexample, 3) superior inspectors may perform a random inspection on theproduct defects. For example, these superior inspectors may extract aplurality of product defect images (for example, 100 product defectimages) for a certain product defect, and then manually judge and labelthe product defects. Subsequently, a plurality of inspectors mayindependently review the inference results of the AI defectidentification model for the same sample (product defect) by using thelabeled product defects. After the end of review, the plurality ofinspectors can vote on inference results of the same product defect, andthe inference result with highest number of votes may be taken as astandard result for the product defect. Subsequently, the standardresult is compared with the inference result of the AI defectidentification model to monitor the accuracy, the precision, the recall,or the FScore of the AI defect identification model.

Third Method: Monitoring Based on Extracting LOT Products

A plurality of (for example, 3) superior inspectors may extract the AIdefect identification model to review the inference results of theproduct defects of the entire LOT (for example, one LOT, where one LOTmay include 20 Glass, and each Glass may include 50 to 300 productdefects) products. The product manufacturing message processing device(103) obtains review results and determines the accuracy, the precision,the recall, or the FScore of the AI defect identification modelaccording to the review results.

The product manufacturing message processing device (103) may also beconfigured to monitor the status of manual resources for defectidentification. For example, in the manufacturing procedure of thefactory, there may be cases where the tasks to be processed cannot beprocessed in time due to the low working efficiency of operators, or,some important alarm information cannot be processed by the operators intime. These cases may bring great loss to production. Thus, the productmanufacturing message processing device (103) may also be configured tomonitor the status of the product manufacturing assisting device (104)for the manual identification of the product defects. For example, theproduct manufacturing message processing device (103) may be configuredto monitor the speed of the manual processing on the product defects,which is pushed to the product manufacturing assisting device (104) (forexample, monitoring whether the operators perform the manualidentification on the product images at normal speed), and determine theworking efficiency and the working status of the operators, therebyoptimizing the allocation of tasks of product defectidentification/detection in combination with the scheduling plan.

In the manufacturing procedure of the factory, the product manufacturingmessage processing device (103) may also monitor the usage efficiency ofcomputing resources for the AI defect identification models. If theinference of defect identification cannot be completely finished byusing almost all the computer resources, at this point, hardwareresources and the like may be required to be dynamically added. Theproduct manufacturing message processing device (103) can monitor theusage efficiency of the computing resources in GPU cluster, includingusages of memory and video memory, to determine whether there areanomalies in GPU resources, etc.

The product manufacturing message processing device (103) may also beconfigured to monitor the alarm status in the factory system. In theprocedure of product defect detection, large-scale defect aggregationmay occur. The product manufacturing message processing device (103) maybe configured to match product defect information with the sections ordepartments that handle the product defect in the factory, and sendalarm information in time to these departments. If the product defectrequires the corresponding department to perform process or productionadjustment on the defect, the product manufacturing message processingdevice (103) may be configured to monitor the alarm information and thestate of the alarm being processed in real-time. Moreover, in the eventof a serious alarm not being handled in time, the product manufacturingmessage processing device (103) may be configured to automaticallyupgrade the alarm level, and meanwhile, send the information of theupgraded alarm level to the higher-level department to urge thecorresponding department to pay attention to and handle the seriousalarm in time.

Optionally, the product manufacturing message processing device (103)may also obtain analysis result data for a plurality of product defectanalysis tasks. Also, the product manufacturing message processingdevice (103) may integrate the aforementioned analysis result data basedon one or more of: product defect type, format of the result data, andway to resolve the problems of product defect.

During the manufacturing procedure, there are hundreds types of variousproduct defects due to the presence of various influencing factors, butnot all of these product defects affect the final quality of theproducts, and not all of the product defects affecting the final qualityof the products can be fixed. The product manufacturing messageprocessing device (103) can integrate the analysis result data toprovide the diverse analysis result data to other devices in a timelymanner, thereby facilitating the provision of alarm information andguidance for subsequent process operations, etc. The productmanufacturing message processing device (103) may integrate the resultsof the judgments of the product defect models with the results of themanual review. The product manufacturing message processing device (103)may also integrate the judgments of the product defect models withprocess rules, etc.

For example, the product manufacturing message processing device (103)may integrate the analysis result data described above according to theproduct defect type. For semiconductor manufacturing processes, thedefect types may include particles, remain, line, hole, splash,electrostatic breakdown (ESD), wrinkle, film color, bubble, and thelike, each type of which belong to a major category. In addition, wherenecessary, each major category is further subdivided into multiple minorcategories, e.g., the major category of particles are divided into P0 toP9, which refer to different forms of dust defects respectively. Theproduct manufacturing message processing device (103) may firstlyintegrate the product defects of the major categories based on thecorrespondence between the minor and major categories described above(which may be many-to-many) in order to determine how to subsequentlyrepair and process these product defects. For example, cutting may berequired for a Short type product defect, while joining may be requiredfor an Open type product defect, etc. The product manufacturing messageprocessing device (103) can also integrate and count information aboutthe product defect type to determine which department or section tocomplete the subsequent processing. If a sudden situation where productdefects of a critical type are generated on a large scale arises, theproduct manufacturing message processing device (103) sends a largeamount of alarm information to the corresponding section. The productmanufacturing message processing device (103) may also continue tointegrate the product defects of minor categories. For example, theproduct defects of a same minor category may exhibit different patternsand need to be processed by different departments (Sputter/PECVD), theproduct manufacturing message processing device (103) may integratedifferent information, classify them, and push them to differentdepartments.

In some embodiments, the product defect analysis tasks also includes atask of analyzing the causes of generating the product defects. Theproduct manufacturing message processing device (103) obtains theproduct defect content (defect type, defect location, defect size, etc.)of the product defects, as well as manufacturing process informationrelated to the generation of the product defect content. Themanufacturing process information related to the generation of theproduct defect content includes, for example, the time of manufacturingthe current product, environmental information (e.g., includingtemperature, humidity, pressure, etc.), operator information, equipmentparameter information, material information, configuration information,etc. The product manufacturing assisting device (104) cleans (removesinvalid and problematic data), integrates (integrates into the requiredstandard data format), data mines and analyzes (e.g., performs dataprocessing on the integrated data in unsupervised modes such asclassification, clustering, feature extraction, dimensionalityreduction, and correlation analysis) the aforementioned information(product defect content, as well as manufacturing procedure informationrelevant to the generation of said product defect content). Then, adefect cause analysis model is obtained. The product manufacturingassisting device (104) receives data generated during the productmanufacturing process and can get the causes of the current productdefects based on the defect cause analysis model described above. Thecauses of the current product defects include, for example, insufficientstrength of the cleaning process in the semiconductor generationprocedure, insufficient corrosion, excessive corrosion, inaccuratematching of raw materials, excessive dust in the cleaning environment,insufficient exposure intensity, excessive exposure intensity, foreignmatter doping, and so on. The product manufacturing message processingdevice (103) can provide corresponding adjustment recommendations forprocess and/or equipment according to the causes of current productdefects.

FIG. 7 is another schematic diagram illustrating the method (200) forprocessing product manufacturing messages according to at least oneembodiment of the present disclosure.

Referring to FIG. 7 , the method (200) for processing productmanufacturing messages may also include all or some of thesub-operations in the operation 240. In the operation 240, the productmanufacturing message processing device (103) may update the AI defectidentification model of the product manufacturing assisting device(104).

Optionally, the operation 240 includes sub-operations 241 to 243.

In sub-operation 241, a product defect analysis task for offline testingof a first AI defect identification model is distributed by the productmanufacturing message processing device (103), to obtain a first productdefect analysis result. Assuming that at this time, a second AI defectidentification model is being used in the fourth product manufacturingassisting device (104-4) to analyze product defects of the product type.The product manufacturing message processing device (103) may use thedata generated by the second AI defect identification model throughinference as a test set to test the first AI defect identification modeloffline, to compare the first AI defect identification model and thesecond AI defect identification model and to determine which one isbetter. Optionally, the inspectors in the factory may also review andlabel the data generated by the second AI defect identification modelthrough inference, and the product manufacturing message processingdevice (103) may then take the reviewed and labeled data as a test setfor further test.

As shown in FIG. 7 , the product manufacturing message processing device(103) initiates offline testing of a newly trained model, and theproduct manufacturing message processing device (103) distributes theproduct defect analysis tasks by sending a fourth product defectanalysis request message to the fifth product manufacturing assistingdevice (104-5), and the fourth product defect analysis request messagecontains the storage path of the first product defect identificationmodel and the storage path of the test set required for offline testing.The fifth product manufacturing assisting device (104-5), after parsingthe fourth product defect analysis request message, may schedule productdefect model testing algorithm to perform the testing of the first AIdefect identification model. After the testing of the first productdefect model is completed, the product manufacturing message processingdevice (103) obtains the first product defect analysis result, whichindicates the accuracy, speed, etc. of analyzing the product defects bythe first product defect model. If the first product defect analysisresult meets a predetermined criterion (e.g., the accuracy of the firstproduct defect analysis result is greater than the accuracy of thesecond AI defect identification model), the inspectors may confirm toperform the next test procedure, or it may proceed directly to operation242. Those of skill in the art should understand that the aforementionedpredetermined criteria may also include a comparison of any one of themetrics such as recall rate, F1-Score, etc., of the first product defectanalysis result with the analysis result produced by the second AIdefect identification model for the same data set, and the predeterminedcriteria does not limited by the present disclosure as long as it candetermine which one of the two AI defect identification models isbetter.

In sub-operation 242, a product defect analysis task for online testingof the first AI defect identification model is distributed by theproduct manufacturing message processing device (103), to obtain asecond product defect analysis result.

The online test of the first AI defect identification model may also bereferred to as the grayscale deployment procedure of the first AI defectidentification model. The product manufacturing message processingdevice (103) may perform an online test on the first defectidentification model by using real-time data generated on theproduction-line of the factory. Both the first AI defect identificationmodel and the second AI defect identification model respectively use thesame real-time data and generate different product defect inferenceresults through inference. At this point, the product defect inferenceresult generated by the first AI defect identification model throughinference is the second product defect analysis result described above.The product manufacturing message processing device (103) may furtheranalyze whether the second product defect analysis result meets thepredetermined criteria (e.g., whether it is better than the productdefect inference result produced by the second AI defect identificationmodel through inference based on the same data). Those of skill in theart should understand that the aforementioned predetermined criteria mayalso include a comparison of any one of the metrics such as recall rate,F1-Score, etc., of the first product defect analysis result with theanalysis result produced by the second AI defect identification modelfor the same data set, and the predetermined criteria does not limitedby the present disclosure as long as it can determine which one of thetwo AI defect identification models is better.

As shown in FIG. 7 , if the first AI defect identification model passesoffline testing, the product manufacturing message processing device(103), after reading the data generated in real time on theproduction-line, copies that data into two copies. The two copies aredistributed to a fourth product manufacturing assisting device (104-4)(e.g., a fifth product defect request message is sent to the fourthproduct manufacturing assisting device (104-4) that performs a productdefect identification inference task using the second AI defectidentification model, and a fifth product manufacturing assisting device(104-5) (e.g., a sixth product defect request message is sent to thefifth product manufacturing assisting device (104-5) that performsonline testing of the first AI defect identification model. The fourthproduct manufacturing assisting device (104-4) and the fifth productmanufacturing assisting device (104-5) may be the above GPU cluster andmay also be other devices capable of executing the analysis or test ofthe AI defect identification model. The fourth product manufacturingassisting device (104-4) and the fifth product manufacturing assistingdevice (104-5) may be the same device and may also be different devices.There is no limitation thereto in the present disclosure.

After the fourth product manufacturing assisting device (104-4) and thefifth product manufacturing assisting device (104-5) receive the productdefect analysis request message sent by the product manufacturingmessage processing device (103) for online testing, the fifth productmanufacturing assisting device (104-5) may schedule an AI online testingalgorithm to perform online testing of the first product defect model(the first AI defect identification model). The online test of the firstAI defect identification model and the inference of the second AI defectidentification model are synchronously performed, and data sources ofboth are consistent. Similarly, the product manufacturing messageprocessing device (103) sends a testing task of the first AI defectidentification model to the fifth product manufacturing assisting device(104-5) in a form of a product defect request message. The fifth productmanufacturing assisting device (104-5) may obtain a storage address forthe data set required for online testing of the first AI defectidentification model and a storage address for the first AI defectidentification model from the product defect request message. The fifthproduct manufacturing assisting device (104-5) may then schedule an AIalgorithm to perform an online testing task of the first AI defectidentification model. The fifth product manufacturing assisting device(104-5) may return a second product defect analysis result to theproduct manufacturing message processing device (103) via a sixthproduct defect analysis response message.

In the case where the first product defect analysis result and secondproduct defect analysis result meet the predetermined criteria, insub-operation 243, a model replacement request (e.g., the seventhproduct defect analysis request message illustrated) is sent by theproduct manufacturing message processing device (103) to the fourthproduct manufacturing assisting device (104-4) to replace the second AIdefect identification model in the fourth product manufacturingassisting device (104-4) with the first AI defect identification model.

After the above three sub-operations, the first AI defect identificationmodel trained by the product manufacturing assisting device (104) can beeasily tested offline, efficiently and visually deployed in grayscale(online tested), and updated throughout the release process.

The grayscale deployment and the online test do not affect the real-timeinference task of the original defect identification model (the secondAI defect identification model) on the production-line, then the normalproduction on the production-line is not affected, and large loss causedby a system shutdown and the like due to the deployment of the newdefect identification model (the first AI defect identification model)using traditional manual test would not happen. Moreover, test data usedin the grayscale deployment procedure is review results in the productmanufacturing message processing device (103) after the review of theinspectors and real-time inferred data generated by the old defectidentification model. Other test data is not used, thereby saving thecost of collecting a large amount of data.

Thus, at least one embodiment of the present disclosure makes reasonableuse of the data generated by the entire automated defect classificationsystem, achieving data recycling and reducing the cost of collectingtraining data and testing data. The product manufacturing messageprocessing device (103) completes offline testing and gray-scaledeployment functions of the new model without affecting the productionline, avoiding the loss of the artificial intelligence defectidentification model due to the need for shutdown for testing andupdating. According to embodiments of the present disclosure, theprocess described above may also be implemented as a computer softwareprogram. For example, embodiments of the present disclosure include acomputer program product comprising a computer program tangiblycontained on a machine-readable medium, the computer program comprisingprogram code for implementing methods described above. According toembodiments of the present disclosure, a computer-readable storagemedium with instructions stored thereon is provided, and when theprograms instructions are executed by a processor, the methods describedabove are implemented.

FIG. 8 is a diagram illustrating a device (800) for processing productmanufacturing messages according to at least one embodiment of thepresent disclosure.

The device (800) for processing product manufacturing messages mayinclude: a monitoring unit 801, configured to monitor a plurality ofproduct manufacturing messages; a scheduling unit 802, configured toestablish a product defect analysis task queue based on the plurality ofproduct manufacturing messages; and a distribution unit 803, configuredto distribute product defect analysis tasks to the product manufacturingassisting devices based on the product defect analysis task queue.

FIG. 9 is a structural diagram illustrating an electronic device 900according to at least one embodiment of the present disclosure.

Referring to FIG. 9 , the electronic device 900 according to at leastone embodiment of the present disclosure may include a processor 901 anda memory 902. Both the processor 901 and the memory 902 may be connectedvia a bus 903. The electronic device 900 may be a tower server, a rackserver (Rack), a blade server (Blade Server), a cabinet server, and thelike.

The processor 901 may perform various actions and processing based on aprogram stored in the memory 902. Specifically, the processor 901 may bean integrated circuit chip with signal processing capability. Saidprocessor may be a general purpose processor, a digital signal processor(DSP), a special purpose integrated circuit (ASIC), an off-the-shelfprogrammable gate array (FPGA) or other programmable logic device, adiscrete gate or transistor logic device, or a discrete hardwarecomponent. Each of the methods, steps, and logic block diagramsdisclosed in the examples of this application may be implemented orperformed. The general-purpose processor may be a microprocessor or theprocessor may also be any conventional processor, etc., and may be ofthe x86 or ARM architecture.

Memory 902 stores computer instructions that implement the above methods200 when the computer instructions are executed by processor 901. Thememory 902 may be volatile or non-volatile memory, or may include bothvolatile and non-volatile memory. The non-volatile memory may beread-only memory (ROM), programmable read-only memory (PROM), erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), or flash memory. Volatile memorymay be random access memory (RAM), which is used as an external cache.By way of illustrative but not limiting illustration, many forms of RAMare available, such as static random access memory (SRAM), dynamicrandom access memory (DRAM), synchronous dynamic random access memory(SDRAM), double data rate synchronous dynamic random access memory(DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM),synchronous Link Dynamic Random Access Memory (SLDRAM) and Direct MemoryBus Random Access Memory (DMBRAM). It should be noted that the memoryfor the methods described herein is intended to include, but not belimited to, these and any other suitable types of memory.

The present disclosure of at least one embodiment of a method forprocessing product manufacturing messages, device, and electronic devicecan improve the efficiency of product manufacturing message processingthroughout the product manufacturing process, so that each deviceinvolved in product defect detection and analysis operates efficiently,facilitating subsequent defect cause finding and analysis, and improvingthe efficiency of product manufacturing.

It is noted that the flowcharts and block diagrams in the accompanyingdrawings illustrate the architecture, functionality, and operation ofpossible implementations of systems, methods, and computer programproducts in accordance with various embodiments of the presentdisclosure. In this regard, each box in the flowchart or block diagrammay represent a module, segment, or portion of code that contains one ormore executable instructions for implementing a specified logicalfunction. It should also be noted that in some of the as-replacedimplementations, the functions labeled in the boxes may also occur in adifferent order than those labeled in the accompanying drawings. Forexample, two successively represented boxes may actually be executed insubstantially parallel, and they may also sometimes be executed in thereverse order, depending on the function involved. It should also benoted that each box in the block diagram and/or flowchart, andcombinations of boxes in the block diagram and/or flowchart, may beimplemented using a dedicated hardware-based system that performs thespecified function or operation, or may be implemented using acombination of dedicated hardware and computer instructions.

In general, various example embodiments of the present disclosure may beimplemented in hardware or proprietary circuitry, software, firmware,logic, or any combination thereof. Some aspects may be implemented inhardware, while other aspects may be implemented in firmware or softwarethat can be executed by a controller, microprocessor, or other computingdevice. When aspects of embodiments of the present disclosure areillustrated or depicted as block diagrams, flowcharts, or representedusing certain other graphics, it will be understood that the boxes,devices, systems, techniques, or methods described herein may beimplemented as non-limiting examples in hardware, software, firmware,dedicated circuitry or logic, general purpose hardware or controllers orother computing devices, or some combination thereof.

The processor in at least one embodiment of the present disclosure maybe an integrated circuit chip with signal processing capability. Saidprocessor may be a general-purpose processor, a digital signal processor(DSP), an application-specific integrated circuit (ASIC), anoff-the-shelf programmable gate array (FPGA) or other programmable logicdevice, a discrete gate or transistor logic device, or a discretehardware component. Various methods, steps, and logic block diagrams ofthe disclosure in the examples of this application may be implemented orperformed. The general-purpose processor may be a microprocessor or theprocessor may also be any conventional processor, etc., and may be ofthe x86 or ARM architecture.

The computer-readable storage medium of at least one embodiment of thepresent disclosure may be volatile memory or non-volatile memory, or mayinclude both volatile and non-volatile memory. The non-volatile memorymay be read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), or flash memory. The volatilememory may be random access memory (RAM), which is used as an externalcache. By way of illustrative but not limiting illustration, many formsof RAM are available, such as static random access memory (SRAM),dynamic random access memory (DRAM), synchronous dynamic random accessmemory (SDRAM), double data rate synchronous dynamic random accessmemory DDRSDRAM), enhanced synchronous dynamic random access memory(ESDRAM), synchronous Linked Dynamic Random Access Memory (SLDRAM) andDirect Memory Bus Random Access Memory (DMBRAM). It should be noted thatthe memory for the systems and methods described herein is intended toinclude, but not be limited to, these and any other suitable types ofmemory.

It should be noted that the flowcharts and block diagrams in theaccompanying drawings illustrate the architecture, functionality, andoperation that may be implemented in systems, methods, and computerprogram products in accordance with various embodiments of the presentdisclosure. In this regard, each box in the flowchart or block diagrammay represent a module, segment, or portion of code that contains one ormore executable instructions for implementing a specified logicalfunction. It should also be noted that, in some implementations assubstitutions, the functions labeled in the boxes may also occur in adifferent order than those labeled in the accompanying drawings. Forexample, two successively represented boxes may actually be executed insubstantially parallel, and they may also sometimes be executed in thereverse order, depending on the function involved. It should also benoted that each box in the block diagram and/or flowchart, andcombinations of boxes in the block diagram and/or flowchart, may beimplemented using a dedicated hardware-based system that performs thespecified function or operation, or may be implemented using acombination of dedicated hardware and computer instructions.

In general, various example embodiments of the present disclosure may beimplemented in hardware or dedicated circuitry, software, firmware,logic, or any combination thereof. Some aspects may be implemented inhardware, while other aspects may be implemented in firmware or softwarethat can be executed by a controller, microprocessor, or other computingdevice. When aspects of embodiments of the present disclosure areillustrated or described as block diagrams, flowcharts, or representedusing certain other graphics, it will be appreciated that the boxes,devices, systems, techniques, or methods described herein may beimplemented as non-limiting examples in hardware, software, firmware,specialized circuitry or logic, general-purpose hardware, or controllersor other computing devices, or some combination thereof.

The example embodiments of the invention described in detail above aremerely illustrative and not limiting. Those of skill in the art shouldunderstand that various modifications and combinations of these examplesor features thereof may be made without departing from the principlesand spirit of the present invention, and that such modifications shouldfall within the scope of the present invention.

1. A method for processing product manufacturing messages, comprising:monitoring a plurality of product manufacturing messages; establishing aproduct defect analysis task queue based on the plurality of productmanufacturing messages; and distributing product defect analysis tasksto product manufacturing assisting devices based on the product defectanalysis task queue, wherein the product defect analysis tasks include atask of identifying product defect content based on a defectidentification model; wherein the product defect content includes anyone or more of: product defect type, product defect location, andproduct defect size.
 2. The method for processing product manufacturingmessages according to claim 1, wherein the plurality of productmanufacturing messages include at least one single-product manufacturingmessage and at least one LOT-products manufacturing message, and themonitoring the plurality of product manufacturing messages comprises:monitoring LOT-products manufacturing messages by interrupt; andmonitoring single-product manufacturing messages by polling.
 3. Themethod for processing product manufacturing messages according to claim1, further comprising: sending registration information to a productmanufacturing message serving device, wherein the registrationinformation includes product manufacturing site information and/or firstproduct information; monitoring a first product manufacturing messagesent from the product manufacturing message serving device, based on theregistration information, wherein the first product manufacturingmessage is associated with the product manufacturing site informationand/or the first product information.
 4. The method for processingproduct manufacturing messages according to claim 3, further comprising:monitoring a second product manufacturing message sent from the productmanufacturing message serving device, wherein the second productmanufacturing message is irrelevant to the registration information;determining whether a list of product manufacturing keywords includes aproduct manufacturing keyword in the second product manufacturingmessage, reserving the second product manufacturing information in acase where the list of product manufacturing keywords includes theproduct manufacturing keyword in the second product manufacturingmessage; and discarding the second product manufacturing information ina case where the list of product manufacturing keywords does not includethe product manufacturing keyword in the second product manufacturingmessage.
 5. The method for processing product manufacturing messagesaccording to claim 1, wherein, the establishing the product defectanalysis task queue based on the plurality of product manufacturingmessages further comprises: establishing the product defect analysistask queue by sorting the product defect analysis tasks based on any oneor more of: the order in which the product manufacturing messages arereceived, priorities of products, and a product scheduling plan.
 6. Themethod for processing product manufacturing messages according to claim1, wherein, the defect identification model includes any one or more of:a feedforward neural network defect identification model, aconvolutional neural network model, a recurrent neural network model,and a generative adversarial network model.
 7. The method for processingproduct manufacturing messages according to claim 1, further comprising:obtaining a plurality of product images based on the plurality ofproduct manufacturing messages; obtaining product defects from theplurality of product images; wherein the obtaining product defects fromthe plurality of product images comprises any one or more of: performingany one or more of: rotation, scaling, color transformation andinterception of a product image of the plurality of product images in acase of an uneven distribution of the quantity of the product defects inthe product image; magnifying an image region in which a product defectis located in a case where the image region is smaller than a firstpredetermined threshold; merging product images corresponding to any twoproduct defects in a case where a similarity between the two productdefects is greater than a second predetermined threshold; and obtaininga frequency domain image of a product image of the plurality of productimages in a case where a similarity between positive samples andnegative samples in the product image is greater than a thirdpredetermined threshold, and adjusting the positive samples or negativesamples based on the frequency domain image; generating the productdefect analysis tasks based on the product defects.
 8. The method forprocessing product manufacturing messages according to claim 1, whereinthe distributing product defect analysis tasks to product manufacturingassisting devices comprises: obtaining a product type and a productdefect analysis task type from the plurality of product manufacturingmessages; generating a product defect analysis request message based onthe product type and the product defect analysis task type.
 9. Themethod for processing product manufacturing messages according to claim8, wherein the product manufacturing assisting devices include a firstproduct manufacturing assisting device and a second productmanufacturing assisting device, wherein the distributing product defectanalysis tasks to product manufacturing assisting devices based on theproduct defect analysis task queue further comprises: determiningwhether an AI defect identification model corresponding to the producttype is present; sending a first product defect analysis request messageto the first product manufacturing assisting device in a case where theAI defect identification model corresponding to the product type is notpresent, wherein the first product defect analysis request messagecomprises the product type, a storage address of product images, aquantity of the plurality of product images, and a task identity fortraining the AI defect identification model; sending a second productdefect analysis request message to the second product manufacturingassisting device in a case where the AI defect identification modelcorresponding to said product type is present, wherein the secondproduct defect analysis request message comprises the product type, astorage address of the product images, and a quantity of the productimages.
 10. The method for processing product manufacturing messagesaccording to claim 9, further comprising: receiving a first productdefect analysis response message sent by the first product manufacturingassisting device, wherein the first product defect analysis responsemessage comprises one or more of: identity, accuracy, and recall of theAI defect identification model; wherein the AI defect identificationmodel is determined based on the product type, the storage address ofthe product images, and the quantity of the product images.
 11. Themethod for processing product manufacturing messages according to claim9, further comprising: receiving a second product defect analysisresponse message sent by the second product manufacturing assistingdevice, wherein the second product defect analysis response messagecomprises one or more of: product image identity, product defectlocation, product defect identity, and repair identity; wherein theproduct defect location, the product defect identity and the repairidentity are determined based on the product type, the storage addressof the product images, and the quantity of the product images.
 12. Themethod for processing product manufacturing messages according to claim1, further comprising: monitoring one or more of: accuracy, precision,recall, F-score, and speed of processing, by the product manufacturingassisting device, the product defect analysis tasks.
 13. The method forprocessing product manufacturing messages according to claim 1, furthercomprising: obtaining analysis result data from a plurality of productdefect analysis tasks; integrating the analysis result data based on oneor more of: the product defect type, a result data format, and a mannerin which problems of product defects are resolved.
 14. The method forprocessing product manufacturing messages according to claim 13, furthercomprising: sending a product defect alert based on the analysis resultdata.
 15. The method for processing product manufacturing messagesaccording to claim 1, further comprising: updating the defectidentification model.
 16. The method for processing productmanufacturing messages according to claim 15, wherein the updating thedefect identification model further comprises: distributing a productdefect analysis task for offline testing of a first defectidentification model to obtain a first product defect analysis result;distributing a product defect analysis task for online testing of thefirst defect identification model to obtain a second product defectanalysis result; in a case where the first defect analysis result andthe second defect analysis result meet a predetermined criteria,distributing a product defect analysis task for replacing the seconddefect identification model in a product manufacturing assisting devicewith the first defect identification model.
 17. An electronic device,comprising: a processor; a memory, storing computer instructions that,when executed by the processor, implement the following steps:monitoring a plurality of product manufacturing messages; establishing aproduct defect analysis task queue based on the plurality of productmanufacturing messages; and distributing product defect analysis tasksto product manufacturing assisting devices based on the product defectanalysis task queue, wherein the product defect analysis tasks include atask of identifying product defect content based on a defectidentification model; wherein the product defect content includes anyone or more of: product defect type, product defect location, andproduct defect size.
 18. A computer-readable storage medium withcomputer instructions stored thereon, when the instructions are executedby a processor, the method according to claim 1 is implemented.
 19. Theelectronic device according to claim 17, wherein, when are executed bythe processor, the computer instructions further implement the followingsteps: sending registration information to a product manufacturingmessage serving device, wherein the registration information includesproduct manufacturing site information and/or first product information;monitoring a first product manufacturing message sent from the productmanufacturing message serving device, based on the registrationinformation, wherein the first product manufacturing message isassociated with the product manufacturing site information and/or thefirst product information.
 20. The electronic device according to claim17, wherein, when are executed by the processor, the computerinstructions further implement the following steps: obtaining aplurality of product images based on the plurality of productmanufacturing messages; obtaining product defects from the plurality ofproduct images; wherein the obtaining product defects from the pluralityof product images comprises any one or more of: performing any one ormore of: rotation, scaling, color transformation and interception of aproduct image of the plurality of product images in a case of an unevendistribution of the quantity of the product defects in the productimage; magnifying an image region in which a product defect is locatedin a case where the image region is smaller than a first predeterminedthreshold; merging product images corresponding to any two productdefects in a case where a similarity between the two product defects isgreater than a second predetermined threshold; and obtaining a frequencydomain image of a product image of the plurality of product images in acase where a similarity between positive samples and negative samples inthe product image is greater than a third predetermined threshold, andadjusting the positive samples or negative samples based on thefrequency domain image; generating the product defect analysis tasksbased on the product defects.